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PIXEL SELECTIVE WHITE BALANCING 

BACKGROUND 

L Field of the Invention : 

The embodiments described herein relate to image processing techniques. In 
5 particular, these embodiments relate to white balancing multicolored image data, 
1. Related Art : 

Because of the tri-stimulus nature of human color perception, to reconstruct a 
color image of a scene it is typically necessary to recover at least three color components 
(typically red, green, and blue or cyan, magenta, and yellow) for each picture element 
10 (pixel) in the image. In such color imaging systems, white balance is a critical factor in 
I p perceived image quality. White balancing is a process of weighting the intensities of the 

n i individual color channels of a composite color system to achieve the greatest fidehty of 

i^'^ the image compared to the original scene. An objective of white balancing is for a white 

"1, object to be imaged with properly proportioned energies in component colors (e.g., red, 

15 green, and blue). 

Red, green, and blue color channels of a typical electronic imaging device may 
not be in balance with one another. This imbalance is primarily due to the effects of 
ambient lighting. Scenes imaged under fluorescent lights may produce image pixel 
responses that are different from pixel responses of the same scenes imaged under 
20 incandescent light or sunlight. Also, although less likely, inaccuracies in the placement of 
color transmissive filters over an imaging sensor, variations in the circuitry of the 
imaging sensor that comprise the individual pixels, or variations in the analog-to-digital 
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conversion circuits (assuming separate A/D circuits are used for each color stream) may 
introduce color imbalance. 

Various methods have been employed to achieve white balance. One method 
involves an inclusion of a white balance sensor (which may separate from, or combined 
with, a primary imaging sensor) within the system. Here, a camera operator typically 
points the white balance sensor at a white reference surface to extract reference color 
information. In another method, the imaging system may attempt to perform white 
balancing based upon image data extracted from a natural scene. 

A system that requires the camera operator to image a white reference area to achieve 
proper white balance can be cumbersome due to the effects of ambient lighting on color 
channel balance. The operator typically performs the manual white balancing operation every 
time the ambient Ught changes. Scene based white balancing provides a more user friendly 
approach by utilizing information extracted from the imaging sensors independent of image 
data generated by imaging a white reference surface. This scene based approach places the 
burden of extracted color information for white balancing on the image processing algorithms 
not on the camera operator. 

One approach to scene based white balance assumes that the average pixel mtensity 
value for each of the color channels throughout any given scene are equivalent. In an imaging 
system with red, green and blue color channels, for example, one channel is typically 
designated as the reference channel and the other two channels are balanced to the reference 
reference channel. The green channel is typically designated as the reference channel due to 
its greater spectral responsivity over the red and blue channels and its location between red 
and blue channels in the visible light spectrum. Two separate gain factors are then computed 
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and applied to the intensity values of the red and blue channel pixels to bring them into 
balance with the reference green channel. 

The assumption that the average pixel intensity value for each of the three channels are 
equal, however, is not always accurate. In a scene where the averages in the imaged object are 
slightly different, this assumption will cause white areas in the resulting image to take on a 
colored tint. Therefore, tiiere is a need for a scene based white balancing system which 
provides a more accurate representation of the colors in an imaged object. 
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SUMMARY 

Briefly, an embodiment of the present invention is directed to a system and 
method of processing data representative of color information extracted from an array of 
pixels in an imaging array. The imaging array may include a plurality of pixels v^hich are 
responsive to photon energy in a distinct spectral region and provides image data. White 
regions in the image may be identified based upon a dispersion of intensities of 
photoexposure at groups of associated pixels where each pixel in a group is associated 
with a distinct one of the plurality spectral regions or color channels. Gain coefficients to 
be applied to intensities of photoexposure in the image for pixels associated with at least 
one of the color channels may be based upon an accumulation of the intensities of 
photoexposure of the pixels associated with the at least one color channel in the white 
regions of the image. In alternative embodiments, a degree of whiteness is associated 
with regions of the image captured in the imaging array. The pixel values in a particular 
pixel region may provide a weighted contribution to the calculation of gain coefficients 
based upon the degree of whiteness associated with that particular pixel region. 
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BRIEF DESCRIPTION OF THE FIGURES 

Figure 1 shows a schematic block diagram of an electronic camera system 
according to an embodiment. 

Figures 2A, 2B and 2C show embodiments of the imaging sensor shown in the 
embodiment of Figure 1 . 

Figure 3 is a functional flow diagram illustrating processes executed at the image 
processor shown in Figure 1 according to an embodiment. 

Figure 4 shows a pattern of pixel data at locations in an image for illustrating a 
process of determining spatially neighboring pixels according to an embodiment. 

Figures 5 through 8 illustrate the process of identifying which pixels are in a 
white region of an image for determining gain factors in the white balance step shown in 
Figure 3, 

Figure 9 shows a functional flow diagram illustrating steps in the process of 
computing gain factors to be used in white balancing multiple color channels in an 
imaging system according to one embodiment. 

Figure 10 illustrates an ahemative process for identifying which pixels are in a 
white region of the image in the white balancing step shown in Figure 3. 

Figure 1 1 illustrates an embodiment in which fuzzy logic is used to determine 
weighting factors in computing white balance gain coefficient based upon degrees of 
whiteness in regions of an image. 
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DETAILED DESCRIPTION 

Embodiments of the present invention are directed to a system and method for 
determining gain coefficients to be applied to image data from multiple color channels in 
an imaging system for carrying out a white balancing process. Upon capturing an image 
over an exposure period, an imaging array provides data representative of intensities of 
photoexposure in distinct spectral regions at specific pixel locations on the imaging array, 
If the intensities of photoexposure of a group of associated pixels of different colors or 
spectral regions are proportionally equivalent, the group of associated pixels are 
determined to be in a "white" region of the image. The gain coefficients are determined 
from the intensity values of the group of associated pixels in the white regions of the 
image. The gain coefficients may then be applied to the pixel data from the multiple 
color charmels to provide white balanced color image data. 

Figure 1 shows a block diagram for an electronic camera system according to an 
embodiment. An imaging sensor 12 is preferably an electronic sensor which receives an 
image focused thereon by an optical system (not shown). The imaging sensor 12 may 
include a semiconductor substrate with transmissive filters deposited over selected 
locations to form pixels. Each pixel is sensitive to photoexposure in a particular spectral 
region defined by the transmissive properties of the transmissive filter deposited over the 
associated pixel location. The semiconductor substrate includes circuitry for extracting 
data which is representative of an intensity of photoexposure of particular pixel elements 
over an exposure period. The semiconductor substrate may be formed as a charge couple 
device (CCD) or an active pixel sensor (APS) as described in U.S. Patent Nos. 5,471,515 
and 5,587,596. 
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Circuitry at section 14 performs functions for automatic gain control, correlated 
double sampling and analog-to-digital conversion functions to provide digital data 
representative of the intensity of the photoexposure of pixel elements at specific locations 
on the imaging sensor 12. For embodiments in which the imaging sensor 12 is of an APS 
5 design, the intensity values representative of the photoexposure of the pixel elements in 
the array may be extracted using pixel readout circuitry described in U.S. patent 
application Serial No. 09/274,424, entitled "Pixel Read-Out Architecture," filed on March 
22, 1999, assigned to Biomorphic VLSI, Inc., and incorporated herein by reference. 

An image processor 18 receives raw digital sensor data captured at the imaging 

10 sensor 12 and transforms the raw digital sensor data into a full color image. For color 

image data defining three colors at each pixel location and representing the pixel intensity 
value for each color at that pixel location with one eight bit value, each pixel location is 
associated with 24 bits of data to define the intensity values at each of the three spectral 
regions associated with the respective three colors in the image. Other embodiments may 

15 represent pixel intensity values with ten, twelve or more bits with an associated multiple 
of bits representing the total color information at the pixel location. Thus, for a three- 
color image, the image processor 1 8 preferably provides three overlapping arrays or sub- 
images of pixel data, each array containing all of the pixel intensity values for an 
associated color channel. 

20 Figure 1 depicts the image sensor 12, circuitry at section 14 and image 

processor 13 as separate functions. In one embodiment, these functions may be 
performed by three corresponding separate integrated circuits. In other embodiments, the 
image sensor 12, circuitry at section 14 and image processor 18 may be integrated into 
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the same semiconductor substrate. In other embodiments, portions of the functionahty of 
the image processor 18 may be formed in the same semiconductor substrate as the image 
sensor 12 and the circuitry at section 14 while other portions of the functionality may be 
formed in a separate semiconductor substrate. 
5 As discussed below with reference to Figure 3, the image processor 1 8 executes 

several processes for transforming the raw digital sensor data into a full color image. 
According to an embodiment, the image processor 1 8 reads from and writes to an image 
buffer 1 6 during intermediate computations and manipulations of the image data. The 
image processor 1 8 may execute the processes for transforming the raw digital sensor 

10 data by any one of several commercially available programmable RISC processors or 
digital signal processors. Alternatively, the image processor 18 may have an application 
specific integrated circuit (ASIC) design. 

A JPEG code 28 may provide data compression. Instructions executed by a 
microcontroller 42 may be stored in a read-only memory (ROM) 36. DRAM 38 or flash 

15 memory 34 may store data representative of images processed at the image processor 18. 
A memory card 30 may then store finished images, serving as an electronic "film," 
through a memory card interface block 32. An LCD display 26 provides a view finder 
while a photographer is taking pictures. A serial interface such as an RS-232 serial port 
or a universal serial bus (USB) (not shown) may couple the electronic camera system 10 

20 to a personal computer system. A television encoder 20 may also couple the electronic 
camera system 10 to a television set 22. 

Figure 2 A shows an embodiment of the imaging sensor 12 which includes pixels 
for extracting color information in red, green and blue spectral regions. The letters R, G 
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and B represent locations of red, green and blue pixels, respectively. As pointed out 
above, pixels may be formed by depositing a transmissive filter over specific pixel 
locations as described in detail in U.S. patent application Serial No. 09/274,413, entitled 
"Color Filter Pattern," filed on March 22, 1999, assigned to Biomorphic VLSI, Inc., 
5 incorporated herein by reference. The color pattern of pixels distributed over the imaging 
sensor 12 is shown as having a typical Bayer pattern which is described in U.S. Patent 
No. 3,971,065. 

Figure 2B shows an alternative embodiment of the imaging sensor 12 with a red, 
green, blue and white color pattem where the letters R, G, B and W represent locations of 

10 red, green, blue and white pixels, respectively. As discussed in the aforementioned U.S. 
patent application Serial No. 09/274,413, the white pixels may be formed in a 
semiconductor imaging device by maintaining an absence of a transmissive filter 
deposited over white pixel locations while associated transmissive filters are deposited 
over the red, green and blue pixel locations. Figure 2C shows an additional embodiment 

15 of the imaging sensor 12 with a four color filter pattem having different colors such as 
cyan, magenta, yellow and white, where the letters C, M, Y and W represent locations of 
cyan, magenta, yellow and white pixels, respectively. 

Figure 3 illustrates a process for providing finished color images from raw digital 
sensor data. Steps 110 through 122 are preferably performed at the image processor 18 

20 (Figure 1). According to an embodiment, the image processor 18 performs the steps 110 
through 122 with logic circuitry formed in an ASIC (e.g., when the image processor 18 is 
formed in the same semiconductor substrate as the circuitry at section 14 and image 
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sensor 12). Such logic circuitry may be formed in an ASIC using Verilog or VHDL 
tools. 

In other embodiments, the image processor 18 performs the steps 1 10 through 112 
by executing algorithms encoded into computer readable instructions. In this 
5 embodiment, the image processor 1 8 may retrieve the computer readable instructions 
from the ROM 36 or DRAM 38 or the computer readable instructions may be stored in a 
programmable ROM (not shown) which is a part of the image processor 18. In other 
embodiments, the image processor 18 may retrieve the computer readable instructions 
from a peripheral memory device such as the flash 34, a hard disk drive (not shown) or a 

10 floppy disk drive (not shown). 

A dark current correction section 104 subtracts dark current components in the 
raw sensor data received from the imaging sensor 12. Invariably, some of the pixels in 
the imaging sensor 12 may be defective. A defect concealment section 106 compensates 
for these defects by, for example, interpolating from neighboring pixels of the same color 

15 or replacing the data from the defective pixel with that of the nearest neighboring non- 
defective pixel of the same color. A defect map is preferably stored in a memory (such as 
the image buffer 16) to maintain a record of a defective pixel in the imaging sensor 12. 

Figure 3 illustrates a process of white balancing at step 110 followed by a process 
of color interpolation at step 112. White balancing at the step 110 includes the 

20 calculation of gain coefficients to be applied to the pixel intensity data from each of the 
color channels to correct for the aforementioned problem of unbalanced color 
information in the image data. Processes for white balancing are discussed below with 
reference to particular embodiments of the present invention. 
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Embodiments of the imaging sensor 12 illustrated in the above mentioned U.S. 
Patent Application Serial No. 09/274,413 receive color intensity information at each pixel 
location data for a distinct color or spectral region. Therefore, at any particular pixel 
location color information is collected for a single color. Color information for the other 

5 color channels is not collected at the particular pixel location. The process of color 
interpolation at step 1 12 provides pixel intensity value for each of the color charmels at 
each pixel location. This may be performed by a process of color selection whereby the 
imaging sensor 12 provides color information for one particular color channel at a 
particular pixel location, and color data extracted at adjacent pixel locations provides the 

10 color information for the other color channels as described in detail in U.S. Patent 
Application Serial No. 09/482,270, entitled "Color Selection for Sparse Color Image 
Reconstruction," filed on January 13, 2000, assigned to Biomorphic VLSI, Inc., and 
incorporated herein by reference. Alternatively, step 112 may perform a color 
interpolation process to provide image data for all colors at a pixel location as described 

15 in U.S. Patent Application Serial No. 09/482,844, entitled "Continuity Condition in Color 
Interpolation," filed on January 13, 2000, assigned to Biomorphic VLSI, Inc., and 
incorporated herein by reference. The color interpolation step 1 12 preferably provides a 
component sub-image for each color channel. 

At step 114, the image processor 18 performs black level flare processing to 

20 remove any bias in the intensity values resulting from, for an example, an overexposure 
due to camera flash. Such black level flare processing may be performed using 
algorithms which are well known in the art. Each pixel of a particular color or spectral 
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region may also be responsive to photon energy which is outside of the spectral region 
associated with the pixel, introducing a color error. At step 116, the image processor 18 
performs color correction to account for any such errors in the pixel data. 

The image processor 1 8 may then perform a gamma correction process at 

5 step 1 1 8 which compensates for non-linearities in display devices. Here, the image data 
may be adjusted for a display onto a CRT device such as a television screen or computer 
monitor using a standard gamma of 2.2, as specified in the ITU-RBTJ09 standard. 

The processes of color interpolation at step 112 and white balance at step 1 10 may 
blur certain edges in the image. Step 120 preferably enhances edges in the image which 

10 may have been blurred in the color interpolation and white balance processing. 

Finally, the image processor 1 8 converts the color channels of the digital image, 
red, green and blue, for example, to a different color space such as YcbCr color space. 
The embodiment of Figure 3 shows the process of white balancing at step 110 occurring 
prior to the process of color interpolation at step 112. Here, the white balancing process 

1 5 at step 1 1 0 provides image data of a single color at each pixel location corresponding 
with the arrangement of pixel colors on the imaging sensor 12. The process of color 
interpolation then converts this image data into data representative of multiple 
overlapping sub-images corresponding to the number of different colors of pixels on the 
imaging sensor 12. Each of these sub-images includes data representative of pixel 

20 intensity at each location in the sub-image for a color or spectral region associated with 
the sub-image. Thus, in the embodiment of the imaging sensor 12 as shown in 
Figure 2 A, the color interpolation process at step 112 will provide data representative of 
three sub-images corresponding to red, green and blue. 
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In an alternative embodiment, the process of color interpolation may precede the 
process of white balancing. Here, immediately following the process of defect 
concealment at step 106, the color interpolation process converts the image data with 
pixel data for a single color at each pixel location into multiple overlapping sub-images. 

5 The process of whhe balancing is then performed over the data representative of all of the 
sub-images provided by the color interpolation process. 

Embodiments of the present invention are directed to a system and method for 
performing the white balancing process at step 1 10 as shown in the embodiment of 
Figure 3 and, in an alternative embodiment, performing white balancing following color 

10 interpolation. Preferred embodiments of the white balancing process determine which 
pixels of the captured image are in a "white" region of the image. Groups of associated 
pixels of the different colors spanning the color space captured at the imaging sensor 12 
having proportionally equivalent energy are identified as being in a white region of the 
image. The white balancing process calculates gain coefficients to be applied to the 

15 image data based upon the average intensity values of the pixels in the white regions of 
the image for each color channel. The calculated gains may then be applied to the image 
data over the entire image to provide a white balanced image. 

In a step of identifying the white regions of the image, the white balancing 
process determines groups of associated pixels and compares the pixel intensity values 

20 collected at the pixel locations of the associated pixels. In the embodiment in which the 
white balancing process follows color interpolation, a group of associated pixels includes 
pixels in the different sub-images at the same overlapping pixel location. Thus, if while 
balancing is performed on overlapping sub-images of red, green and blue pixels, each 
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group of associated pixels will include a red pixel selected from a red sub-image, a green 
pixel selected from a green sub-image and a blue pixel selected from a blue sub-image, 
all being at the same overlapping pixel location in their respective sub-images. In the 
embodiment in which the white balancing process precedes the color interpolation 

5 process, as shown in Figure 3, the white balancing process at step 110 determines each 
group of associated pixels as spatially neighboring pixels by selecting pixels which are 
adjacent to pixels of a reference color channel 

Step 1 10 may process color image data stored in one or more arrays in a memory 
associated with the image processor 18 (Figure 1) to provide white balanced color image 

10 data. Hence, step 110 of this embodiment stores the white balanced color image having 
data representative of an intensity of one or more spectral regions associated with each 
pixel location in the imaging sensor 12 data in the memory for further processing. In 
other embodiments, step 110 may process image data in a streamed fashion by, for 
example, processing the image in sets of pixel lines or as a processing stage in a pipeline 

15 process. Here, step 1 10 may store in a memory, at any one time, only that image data 
required to process a portion of the image. The remaining image data is processed and 
stored during earlier or later stages. 

Figure 4 shows a representation of digital image data provided by the process for 
defect concealment at step 106 (Figure 3) received at the white balancing process at 

20 step 1 10. The letters R, G and B identify the specific color channel corresponding to 
each pixel location in a Bayer pattern. There are twice as many green pixels as there are 
red or blue pixels. The reference channel is preferably chosen as green since green has a 
greater spectral responsivity over the red and blue, and is located between red and blue in 
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the visible light spectrum. Alternatively, either the red channel or blue channel may 
serve as the reference channel. 

The pixels in the data mapping 150 may be associated into sets of spatially 
neighboring pixels using any one of several approaches. For example, the pixels may be 
5 initially grouped with two green pixels, a red pixel and a blue pixel as shown in 

grouping 152. The pixel intensity values at the green pixel locations are then compared 
with the intensity values at the red and blue pixel locations to determine whether the four 
grouped pixels are in a white spectral region. This is accomplished by determining 
whether the pixel intensity values are proportionally equivalent as discussed below. In 

10 one embodiment, the pixel intensity values at two green pixel locations in grouping 152 
are first averaged and then compared with the pixel intensity values of the red and blue 
pixels in the grouping 152. Alternatively, the four pixels in the grouping 152 are formed 
into two sub-groupings, each sub-grouping having the pixel data for one of the green 
pixels in the grouping 1 52 and the pixel intensity values of each of the red and blue pixels 

15 in the grouping 152. The remaining pixel data shown in Figure 4 may then be partitioned 
into groups of four pixels and illustrated by grouping 152 for determining spatially 
neighboring pixels making up a group of associated pixels. 

As illustrated in alternative grouping 1 54, the pixel intensity values for each green 
pixel may be initially grouped with the pixel intensity values for each of the adjacent 

20 pairs of red and blue pixels. In one embodiment, the pixel intensity values of the pair of 
blue pixels in the grouping 154 are averaged and the pixel intensity values of the pair of 
red pixels are averaged. These averaged pixel intensity values then form the basis of 
comparison of the intensity values of the spatially neighboring pixels in the grouping 154. 
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Alternatively, four sub-groupings of spatially neighboring pixels may be formed from the 
grouping 154 by associating the green pixel in every combination of red and blue pixels 
in the grouping 154. This analysis can then be repeated for determining sets of spatially 
neighboring pixels corresponding to each of the green pixels as described with reference 

5 to the grouping 1 54. While groupings 1 52 and 1 54 illustrate two particular methods for 
determining spatially neighboring pixels in the pixel data mapping 150, it is understood 
that other groupings can be used to determine spatially neighboring pixels in the white 
balancing process v^thout departing from the embodiments of the present invention. 
The analog-to-digital conversion circuitry at section 14 (Figure 1) preferably 

10 quantizes the exposure intensity values for each of the pixels in the imaging sensor 12 
into fixed length binary words as discussed in the aforementioned U.S. patent application 
Serial No. 09/274,424, In an embodiment in which the intensity value for each pixel is 
quantized into eight bit words, the pixel intensity values at each pixel location have one 
of 256 discrete values from 0-255. Other embodiments may provide pixel values 

15 quantized into ten bits or twelve bits to allow each pixel intensity value to assume 

additional values. The image processor 18 may maintain this quantization throughout the 
processing steps shown in Figure 3. However, the process of white balancing described 
herein is also applicable to systems in which the quantization of the pixel intensity values 
changes from that provided by the initial analog-to-digital conversion. 

20 Figures 5 through 8 illustrate a process of determining whether a group of 

associated pixels are in a white region of the image captured at the imaging sensor 12. 
The dark arrows on Figures 5 through 8 identify color component pixel intensity values 
for the group of associated pixels on a scale of 0-255, corresponding with eight bit pixel 
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intensity values. In this embodiment, pixels from three color channels, red, green and 
blue are being white balanced. However, it is understood that this process may also be 
applied to identifying which groups of associated pixels are in a white region in a 
different color space having more than three color channels or colors other than red, 
green and blue. 

In an initial step of the process for determining whether a group of associated 
pixels are in a white region of the image, the intensity value of the pixel from the 
reference channel, being green in the presently illustrated embodiment, is preferably 
within a minimum and maximum intensity value defining a reference green window. If a 
green pixel in a group of associated pixels has a pixel intensity value which is outside of 
the reference green window, the group of associated pixels is not considered to be in a 
white region of the image. This disqualifies groups of associated pixels (from being in a 
white region of the image) whose green response is inaccurate due to limitations of a 
dynamic range of the imaging sensor 12. For example, pixels whose green component is 
too bright may be artificially white and skew the white balance gain factors toward green, 
giving the resulting image a greenish tint. 

In the illustrated embodiment, the reference green window is defined as lying 
between the values 127 and 217. However, the reference green window may be defined 
by other minimum and maximum values. Figures 5, 7 and 8 illustrate cases in which the 
green pixel of a group of associated pixels falls within the reference green window and 
Figure 6 illustrates a case in which the pixel intensity value of a green pixel exceeds the 
upper bound of the reference green window. 
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Following a determination that the pixel intensity value of a green pixel is within 
the reference green window, pixel intensity values of the associated red and blue pixels 
are compared with the intensity value of the green pixel As shown in Figures 5 through 
8, a calculated red/blue window is determined based upon the intensity value of the green 

5 pixel. The calculated red/blue window preferably identifies red and blue pixel values 
which provide a ratio with the pixel value associated with the reference green pixel 
within a desired range. In the embodiment illustrated in Figures 5 through 8, the 
calculated red/blue window is defined as a region centered about the green pixel intensity 
value and extending +/- 45% of the green pixel intensity value. For example, with a 

10 green pixel intensity value of 1 87, the calculated red/blue window will extend from the 
value of 1 03 to 271. The red and blue pixel intensity values are within a calculated 
red/blue window in the cases illustrated in Figures 5 and 8. 

The specific values used, including the window sizes and placements, in the 
example above in connection with Figures 5 through 8 are merely provided for 

15 illustration purposes. It should be understood that other values can be selected based 
upon the specific characteristics of the imaging sensor being used. 

For those groups of associated pixels in which 1) the green pixel intensity values 
are within the reference green window and 2) the red and blue pixel intensity values are 
within the calculated red/blue window, the closeness of the red and blue pixel intensity 

20 values is evaluated. According to an embodiment, the group of associated pixels are 
further evaluated to determine whether the difference between the red and blue pixel 
intensity values is less than 30% of the entire range. In the embodiment illustrated in 
Figures 5 through 8, the red and blue pixel intensity values must be less than 77 apart. 

18 
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Figure 5 illustrates a case in which the pixel intensity values of a group of associated 
pixels meet all three criteria for being selected as being in a white region of the image: 
the green pixel intensity value is within the green reference window; the red and blue 
pixel intensity values are within the calculated red/blue window; and the difference 
5 between the red and blue pixel intensity values is no greater than 30% of the entire range 
from 0 to 255. 

Figure 6 illustrates a case in which the green pixel intensity value is outside the 
green reference window and the red and blue pixel intensity values are outside the 
calculated red/blue window. Figure 7 illustrates a case in which the green pixel intensity 

10 value is within the green reference window, but the red and blue pixel intensity values are 
outside of the calculated red/blue window. Figure 8 illustrates a case in which the green 
pixel intensity value is within the green reference window and the red and blue pixel 
intensities are within the calculated red/blue window. However, Figure 8 shows that the 
red and blue pixel intensity values are more than 77 (or 30% of the entire range from 0 to 

15 255) apart. Therefore, the group of associated pixels shown in Figure 8 are not identified 
as being in a white region of the captured image. 

Figures 5 through 8 illustrate one particular technique for determining whether a 
group of associated pixels are in a white region by evaluating a dispersion of the pixel 
intensity values. It should be understood by those of ordinary skill in the art that other 

20 techniques may be employed for evaluating a dispersion of the pixel intensity values of 
groups of associated pixels to identify pixel intensity values which are proportionally 
equivalent and indicative of a white region of the image. 
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With the green channel being the reference channel in the presently illustrated 
embodiment, white balancing is performed on the remaining red and blue color channels 
relative to the reference green channel. Gains are applied to the intensity values of all red 
and blue pixels in the image (i.e., all pixels not belonging to the reference color channel) 
as follows: 

Rout(iJ) = R(iJ)^Rbai (1) 
Bout(iJ)-B(iJ)'^Bb^ (2) 
Where: 

• Rout(i, j) is the white balanced pixel intensity value for 
the red pixel at the location ith row and jth column location in 
the image; 

• Bout(i, j) is the white balanced pixel intensity value for 
the blue pixel at the location ith row and jth column location in 
the image; 

• R(i J) is the pre-white balanced pixel intensity value 
for the red pixel at the location ith row and jth column location 
in the image; 

• B(i, j) is the pre-white balanced pixel intensity value 
for the blue pixel at the location ith row and jth column 
location in the image; 
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• Rbaj is the white balancing gain coefficient apphed to 
the pre-white balanced pixel intensity values of all red pixels 
in the image; and 

• Bbai is the white balancing gain coefficient applied to 
5 the pre-white balanced pixel intensity values of all blue pixels 

in the image. 

According to an embodiment, the white balancing gain coefficients Rbai and Bbai are 
determined as follows: 

Rbal =^ Gwavg / Rwavg 0) 
1 0 Bbai = Gwavg / Bwavg (4) 

Where: 

• Gwavg is the average pixel intensity value of the green 
pixels (i.e., pixels of the reference channel) which are 
determined to be in the "white'' regions of the image; 

15 • Rwavg is the average pixel intensity value of the red 

pixels which are determined to be in the "white" regions of the 
image; and 

• Bwavg is the average pixel intensity value of the blue 
pixels which are determined to be in the "white" regions of the 

20 image. 

The values of Rwavg, Gwavg and Bwavg may be determined using any of several 
different techniques applying the principle of determining which pixels are in a white region 
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of the image, and averaging the intensity values in each channel over all pixels in the white 
region. In the illustrated embodiment, Rwavg, Gwavg and Bwavg are determined as follows: 

Rwavg^ Rwacc / Rwcount (5) 
Gwavg- Gwacc / Gwcount (6) 

(7) 

Where: 

• Rwacc is the accumulation of the pixel intensity values 
for all red pixels in the "white" regions of the image; 

• Rwcount is the number of red pixels in the "white" 
regions of the image; 

• Gwacc is the accumulation of the pixel intensity values 
for all green pixels in the "white" regions of the image; 

• Gwcount is the number of green pixels in the "white" 
regions of the image; 

• Bwacc is the accumulation of the pixel intensity values 
for all blue pixels in the white regions of the image; and 

• Bwcount is the number of blue pixels in the white 
regions of the image. 

When the image data is represented in separate component sub-images in which each 
component sub-image defines a pixel intensity value at each pixel location for a color channel 
associated with the sub-image (e.g., when the white balancing process is performed after a 
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process of color interpolation), the values of Rwacc, Gwacc Bwacc J^ay be determined by 
accumulating the pixel intensity values of all pixels of the corresponding sub-image which are 
in a white region of the image. It therefore follows that the values Rwcount? Gwcount and Bwcoum 
are determined by counting the number of pixels in the corresponding sub-image which are in 
5 a white region of the image. 

In the embodiment in which white balancing is performed on pixel intensity data 
represented as single color pixel intensity values at each pixel location in the image (e.g., 
when the white balancing process is performed on image data extracted from a Bayer pattern 
imaging array prior to a process of color interpolation), the values of Rwacc, Gwacc and Bwacc 
10 may be determined by separately accumulating for each color channel the pixel intensity 
values of all pixels of the corresponding of the corresponding color which are in a white 
region of the image. It therefore follows that the values Rwcount. Gwcount and Bwcoimt are 
determined by counting the number of pixels of the corresponding color channel which are in 
a white region of the image. 

1 5 The values of Rwacc, Gwacc, Bwacc, Rwcounb Gwcount and Bwcount niay be calculated 

according to the following steps of pseudo code: 

100 Rwacc^O 
110 Gwacc =0 
120 Bwacc~0 

20 130 initialize Ghigh 
140 initialize Gtow 
1 50 initialize RBthreshoid 
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1 60 determine K groups sets of associated R, G and B pixels of the image data 
1 70 for every group of associated pixels from f = 1 to K, 



1 80 if (Git) > Glow) and (G(«) < Ghigh) 

1 90 compute RBhigh from G(t) 

5 200 compute RBiow from G(() 

210 if (R(«) > RBiow) and (R(«) < RBhigh) and (B(e) > RBiow) and (B(t) < RBhigh) 
220 if (abs(R(f) - B(f)) < RBtehoid) 

230 Rwacc ~ Rwacc R(^) 

240 Rwcount-Rwcount+ 1 

1 0 250 Gwacc Gwacc + G(t) 

260 Gwcount = Gwcount+ 1 

270 Bwacc ~ Bwacc 

280 Bwcount-Bwcount+ 1 

290 endif; 

15 310 endif; 



320 endif; 
330 Nexti 

Steps 100 to 150 initialize values which determine the average intensity values of 
the red, green and blue pixels which are determined to be in a white region of the image 
20 according to equations (5), (6) and (7). Step 160 determines groups of associated pixels 
for performing a comparison of intensity values. In the embodiment in which white 
balancing is performed on pixel intensity data of a composite of sub-images (e.g., 
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following the color interpolation at step 112), step 160 forms each group of associated 
pixels by selecting one pixel in each sub-image from the same pixel location. In the 
embodiment in which white balancing is performed on pixel intensity represented as 
single color pixel intensity values at each pixel location in the image (e.g., when the 
5 white balancing process is performed on image data extracted from a Bayer pattern 

imaging array prior to a process of color interpolation), step 160 may form the groups of 
associated pixels as discussed above with reference to Figure 4. 

Steps 1 80 through 200 initialize the reference green window and the calculated 
red/blue window as discussed above with reference to Figures 5 through 8. Steps 210 

10 through 280 determine whether pixels in the groups of associated pixels are in white 
regions of the image, and then adjust the values used to determine the average pixel 
intensity values of the color chaimels. In particular, step 1 80 determines whether a green 
pixel in the group I has an associated pixel intensity value within the reference green 
window. Steps 180 and 190 determine the red/blue window and step 200 determines 

15 whether the red and blue pixel intensity values in the group t are within the red/blue 
window. Step 210 determines whether the pixel intensity values of the red and blue 
pixels in the group t are similar enough to determine that the pixels in the group of 
associated pixels are in a white region of the image. If so, the pixel intensity values of 
the red, green and blue pixels are added to the accumulation of the intensities of the red, 

20 green and blue pixels in the white regions of the image. 

Figure 9 shows a functional flow diagram illustrating a process of determining the 
inputs for determining white balancing gain coefficients in an embodiment in which the 
pixel intensity data represents separate component sub-images for each color. Steps 308 
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through 316 determine whether the pixels in the group of associated pixels I are in white 
regions of the image. Step 3 1 8 aggregates the pixel intensity values of the red, green and 
blue pixels in the white regions of the image, and determines counts of the red, green and 
blue pixels in the white regions of the image. These values may then be used in 
5 calculating the white balancing gain coefficients according to equations (3) through (7). 

Figure 10 illustrates an alternative embodiment for determining whether a group 
of associated pixels are in a white region of the image. Like the embodiment illustrated 
above with reference to Figure 5, the embodiment of Figure 10 first determines whether a 
pixel intensity value of a reference channel (the reference channel being green in the 

10 presently illustrated embodiment) is within a predetermined reference window. If this 
condition is satisfied, a window corresponding to one of the non-reference channels, red 
for example, is calculated based upon the intensity value of the associated pixel in the 
spectral region associated with the reference channel. If the intensity value associated 
with the non-referenced red channel is within the calculated red window, a third window 

15 is calculated for evaluation of the pixel value associated with the remaining channel 
(which is blue in the presently illustrated embodiment) based upon either the reference 
channel pixel value, the red pixel intensity value or both. If the intensity value associated 
with the pixel of the remaining color channel is within this third window, it is determined 
that this group of associated pixels is in a white region of the image. Upon this 

20 determination, the intensity values of the group of associated pixels are accumulated as 
illustrated in steps of pseudocode 230 through 280. 

The above-described embodiments apply the white balancing coefficients 
calculated above in equations (3) and (4) to pre-white balanced intensity values for red 
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and blue pixels in the image. In an alternative embodiment, the white balance gain 
coefficents calculated in equations (3) and (4) are used for selecting coefficients from a 
set of pre-computed white balancing coefficients. One of the objectives of a white 
balancing process is to respond to various lighting scenarios (incandescent, fluorescent, 
5 sunlight, etc.). In this embodiment, a set of gain coefficients are pre-computed for each 
anticipated lighting scenario and stored in memory such as the DRAM 38 or flash 34 
(Figure 1). The white balancing coefficients for the non-reference color channels are 
calculated as discussed above. The white balancing step 110 (Figure 3) then selects the 
set of pre-computed gain coefficients that most closely matches the white balance 

10 coefficients for calculated based upon the image data. This ensures that an appropriate 
set of white balancing coefficients is always applied to the image data. 

The embodiments described above are directed to performing white balancing on 
image data in three component color channels: red; green and blue. However, other 
embodiments may determine white balancing coefficients for image data in four or more 

15 component color chaimels for data extracted from imaging arrays such as those described 
above with reference to Figures 2B and 2C. Determining white balancing gain 
coefficients based upon pixel intensity values extracted from pixels in the white regions 
of an image remains as an objective. 

Applying the subject white balancing technique to image data received from an 

20 RGBW imaging array such as that shown in Figure 2B, the green channel may still be 
chosen as the reference channel. Alternatively, the white channel may be chosen as the 
reference channel. As with the three color embodiments discussed above, pixels are 
associated into groups of pixels spanning the color space. In the four color embodiment, 
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each group of associated pixels preferably includes four pixels, each pixel providing a 
pixel intensity value representative of an intensity of photoexposure in a spectral region 
associated with one of the four color channels. 

The pixel intensity values of the respective pixels in each group of associated 
5 pixels may then be used to compute windows as illustrated in Figures 5 through 8 to 
determine whether the group of associated pixels are in a white region of the image. 
Gain coefficients may then be calculated based upon these the pixel intensity values of 
pixels in white regions of the image. 

The embodiments illustrated above are directed to identifying white regions of an 
rl 10 image captured on an array of pixel elements based upon pre-determined relationships 
\n (e.g., ratios of intensity values) among pixels forming white regions within the image, 

111 and determining white balancing gain coefScients to be applied to pixels in all regions of 

^ the image. In alternative embodiments, white balance gain coefficients may be 

determined according to fuzzy logic rules which associate degrees of whiteness with 
K 15 regions in the image based on both pre-determined relationships (e.g., ratios of intensity 
:5 values) among pixels and leamed relationships among pixels. The pixels of each image 

region and its associated degree of whiteness may then provide a weighted contribution 
to the determination of the white balance gain coefficients. 

Figure 1 1 depicts a process 400 of employing fuzzy logic rules to determine white 
20 balance gain coefficients according to an embodiment. Fuzzy rules 404a through 404e 
evaluate a set of neighboring pixels 402 in a region of the image (e.g., RG and B pixels 
spanning a color space for the imaging sensor). The fuzzy rules 404a through 404e 
associate the set of neighboring pixels 402 with a fuzzy set 406 where each fuzzy set 406 



1 \Common DataXN^y Fiies\wpd\BIOMORPH\249452VPTO\uspap DOC 



corresponds with a degree of whiteness in the image. An appropriate "defuzzification 
weight" 408 may then be appUed to intensity values of the pixels in the set of neighboring 
pixels 402 based upon the membership of the set of neighboring pixels 402 in a fuzzy set 
406 (i.e., associated degree of whiteness in the image). Several sets of neighboring pixels 
402 in the image, each weighted by an appropriate defuzzification weight 408, may then 
contribute to the determination of white balance gain coefficients 410. 

For an RGB pixel array, white balance gain coefficients determined according to 
the fuzzy logic of Figure 1 1 may still applied to the pre-white balanced intensity values 
for red and blue as shown in equations (1) and (2). However, the gains Rbai and Bbai niay 
be calculated differently as follows: 



Rba] = Gp/Ra 



(8) 



Bbai - Gp/By 



(9) 



where: 



Gp is a weighted average pixel intensity value of the green 



pixels in the image; 



R<x is a weighted average pixel intensity value of the red 



pixels in the image; and 



By is a weighted average pixel intensity value of the blue 



pixels in the image. 
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The weighted average pixel intensity values for a color channel are determined by 
weighting the pixel intensity values of the pixels in a color channel according to the 
application of defuzzification weights according to a fuzzy function as illustrated with 
reference to Figure 11. Equations (10) through (18) below illustrate one particular 
5 example of determining the weighted average pixel intensity values according to fuzzy 
logic. While equations (10) through (18) illustrate one particular embodiment, it should 
be understood that other techniques of determining weighted the average pixel intensity 
values according to fuzzy logic may also be used. 

Rcount 

10 Z Ri * tti 

i=l 

Ra= (10) 

Rcount 
S ai 

15 i== 1 

Gcount 

2 Gi * Pi 

1 = l 

20 Gp- (11) 

Gcount 

2 pi 
i= 1 

25 B count 

E Bi * Yi 
i = l 

By= (12) 

Bcount 

30 Syi 
i = l 

ai = fa(Ri,Gi,Bi) (13) 
yi = f,(Ri,Gi,Bi) (14) 
35 Pi = fp(Ri,Gi,Bi) (15) 
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where: 

fa (Ri, Gi, Bi), fp (Ri, Gi, Bi) and f, (Ri, Gi, BO are fuzzy 
functions between zero and one based upon a degree whiteness in a 
region of the image including the group of associated pixels Rj, Gi 
and Bi. 

In one embodiment, one or more of the fuzzy functions fa (Ri, Gi, Bi), fp (Ri, Gi, 
Bi) and f^ (Ri, Gi, Bi) are based upon a dispersion of the intensity values Ri, Gi, and Bi 
about a weighted average of these intensity values as follows: 

fa(Ri,Gi,Bi)-fa((pi) (16) 

fp(Ri,Gi,Bi)-fp((Pi) (17) 
f,(Ri,Gi,Bi) = f,((pi) (18) 
where: 

cpi = (a^i - ^i)' + (Gi - ^li)' + (c^Bi - ^li)' 

• |ii = (a*Ri + Gi + c'^Bi)/(l + a + c); and 

• a and c are constants for scaling red and blue pixel intensity 
values to equivalent pixel intensities for green pixels. 

According to an embodiment, the argument cpi is indicative of a dispersion about 
l^i, a weighted average of Ri, Gi, Bi. The fuzzy functions fa (cpi ), fp (cpi ) and fy (9i ) may 
then be implemented in look-up tables associating ranges of cpi with values from zero to 
one. Such an algorithm for performing white balancing may be implemented in logic 
similar to that illustrated in the flow diagram of Figure 9 by associating groups of pixels, 
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determining a, p and y for each associated group of pixels and determining Rbai and Bbai 
based upon Ra, Gp and By. 

The embodiment illustrated above with reference to equations (10) through (1 8) is 
directed to determining Ra, Gp and By from a weighted sum of all pixels in an imaging 
sensor. In other embodiments, the sets of neighboring pixels 402 (Figure 11) may be pre- 
filtered or pre-selected to be in a white region of the image using techniques such as those 
illustrated above with reference to Figures 5 through 8. The intensity values from each of 
the pre-fihered or pre-selected sets of neighboring pixels may then be weighted according 
to a degree of whiteness associated with the pre-filtered or pre-selected sets of 
neighboring pixels in the determination of Ra, Gp and By. 

While the description above refers to particular embodiments of the present 
invention, it will be understood that many modifications may be made without departing 
from the spirit thereof The accompanying claims are intended to cover such 
modifications as would fall within the true scope and spirit of the present invention. 

The presently disclosed embodiments are therefore to be considered in all respects 
as illustrative and not restrictive, the scope of the invention being indicated by the 
appended claims, rather than the foregoing description, and all changes which come 
within the meaning and range of equivalency of the claims are therefore intended to be 
embraced therein. 
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CLAIMS 

What is claimed is: 

1 . A method of processing data representative of color information extracted from 
an array of pixels in an imaging array, the imaging array including a plurality of pixels 
5 which are responsive to photon energy in a distinct spectral region, each of the pixels 
being capable of outputting an intensity value which is representative of an intensity of 
photoexposure in the spectral region associated with the pixel over an exposure period, 
the method comprising: 

identifying each pixel responsive to photoexposure in a first spectral 
10 region having an intensity value between a minimum intensity value and a maximum 
intensity value to provide a plurality of first pixels; 
for each of the first pixels, 

selecting at least one pixel associated with a second spectral region 
to determine at least one associated second pixel and selecting at least one 
15 pixel associated with a third spectral region to determine at least one 

associated third pixel; and 

associating the intensity value of the associated second pixel and 
the intensity value of the associated third pixel with the intensity value of 
the first pixel to determine a matching set therewith, the matching set 
20 including an associated selected first pixel, the selected second pixel and 

the selected third pixel; and 

determining a first gain coefficient for application to the intensity values 
of each of the pixels associated with the second spectral region based upon an 



I \Comnion DataVNagy Files\wpd\B10MORPH\249452\PTO\uspap DOC 



33 



accumulation of the intensity values associated with the selected second pixels 
and determining a second gain coefficient for application to the intensity values of 
each of the pixels associated with the third spectral region based upon an 
accumulation of the intensity values associated with the selected third pixels. 

5 

2. The method of claim 1 , wherein the step of associating the intensity value of the 
associated second pixel and the intensity value of the associated third pixel with the 
intensity value of the selected first pixel fiirther includes: 

determining whether the intensity value of the associated second pixel is 
10 within a first range of the intensity value of the selected first pixel; and 

determining whether the intensity value of the associated third pixel is 
within a second range of the intensity value of the associated second pixel. 



3. The method of claim 1 , wherein the step of determining the first and second gain 
15 coefficients further includes: 

determining the first gain coefficent as being proportional to an average 
intensity value of all of the selected second pixels divided by an average intensity value 
of each of the selected first pixels; and 

determining the second gain coefficient as being proportional to an 
20 average intensity value of all of the selected third pixels divided by the average intensity 
value of each of the selected first pixels. 
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4. The method of claim 1 , the method further including: 

scaling each of the pixels associated with second spectral region by the 
first gain coefficient; and 

scaling each of the pixels associated with the third spectral region by the 
second gain coefficient, 

5. The method of claim 1 , wherein each of the pixels is associated with a location on 
the imaging array, and wherein the selecting step further includes: 

selecting the at least one second associated pixel as having the same 
location as the first pixel; and 

selecting the at least one third associated pixel as having the same location 
as the first pixel. 

6. The method of claim 1, wherein each of the pixels is associated with a location on 
the imaging array, and wherein the selecting step further includes: 

selecting the at least one second associated pixel as having a first adjacent 
location to the location of the first pixel; and 

selecting the at least one third associated pixel as having a second adjacent 
location to the location of the display. 



I \Common DataNNagy FiIesWpd\BIOMORPH\249452\PTO\uspap DOC 



35 



7, The method of claim 6, the method further including: 

scaling each of the pixels associated with second spectral region by the 
first gain coefficient to provide a first scaled intensity value; 
5 scaling each of the pixels associated with the third spectral region by the 

second gain coefficient to provide a second scaled intensity value; 

determining an intensity value of a pixel associated with the second 
spectral region at the location of the first pixel based upon the first scaled intensity 
value; and 

10 determining an intensity value of a pixel associated with the third spectral 

region at the location of the first pixel based upon the second scaled intensity value. 

8. In a camera, the camera having an imaging array, the imaging array including a 
plurality of pixels which are responsive to photon energy in a distinct spectral region, 

15 each of the pixels being capable of outputting an intensity value which is representative 
of an intensity of photoexposure in the spectral region associated with the pixel over an 
exposure period, a lens for focusing an image of an object onto the imaging array, and a 
processor, the improvement including: 

logic for identifying all pixels responsive to photoexposure in a first 
20 spectral region having an intensity value between a minimum intensity value and a 
maximum intensity value to provide a plurality of first pixels; 

logic for selecting for each of the first pixels at least one associated second 
pixel responsive to photoexposure in a second spectral region; 
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logic for selecting for each of the first pixels at least one associated third 
pixel responsive to photoexposure in a third spectral region; and 

logic for associating the intensity value of the associated second pixel and 
the intensity value of the associated third pixel with the intensity value of the first pixel to 
5 determine a matching set therewith, the matching set including an associated selected 
first pixel, selected second pixel and selected third pixel; and 

logic for determining a first gain coefficient for application to the intensity 
values of each of the pixels associated with the second spectral region based upon an 
accumulation of the intensity values associated with the selected second pixels and for 
10 determining a second gain coefficient for application to the intensity values of each of the 
pixels associated with the third spectral region based upon an accumulation of the 
intensity values associated with the selected third pixels. 

9. A computer readable medium for use in conjunction with an imaging array, the 
15 imaging array having a plurality of pixels which are responsive to photon energy in a 
distinct spectral region, each of the pixels being capable of outputting an intensity value 
which is representative of an intensity of photoexposure in the spectral region associated 
with the pixel over an exposure period, the computer readable medium having computer 
readable instructions encoded thereon for performing the following: 
20 identifying all pixels responsive to photoexposure in a first spectral region 

having an intensity value between a minimum intensity value and a maximum intensity 
value to provide a plurality of first pixels; 

for each of the first pixels, 
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selecting at least one pixel responsive to photoexposure in a second 
spectral region, the second spectral region being distinct from the first 
spectral region, to determine at least one associated second pixel and 
selecting at least one pixel responsive to photoexposure in a third spectral 
5 region, the third spectral region being distinct from the first and second 

spectral regions, to determine at least one associated third pixel; and 

associating the intensity value of the associated second pixel and 
the intensity value of the associated third pixel with the intensity value of 
the first pixel to determine a matching set therev^ith, the matching set 
10 including an associated selected first pixel, selected second pixel and 

selected third pixel; and 

determining a first gain coefficient for application to the intensity values 
of each of the pixels associated v^ith the second spectral region based upon an 
accumulation of the intensity values associated with the selected second pixels and 
15 determining a second gain coefficient for application to the intensity values of each of the 
pixels associated with the third spectral region based upon an accumulation of the 
intensity values associated with the selected third pixels. 



10. An image processor for use in conjunction with an imaging array, the imaging 
20 array including a plurality of pixels which are responsive to photon energy in a distinct 
spectral region, each of the pixels being capable of outputting an intensity value which is 
representative of an intensity of photoexposure in the spectral region associated with the 
pixel over an exposure period, the image processor comprising: 
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logic for identifying all pixels responsive to photoexposure in a first 
spectral region having an intensity value between a minimum intensity value and a 
maximum intensity value to provide a plurality of first pixels; 

logic for selecting for each of the first pixels at least one pixel associated 
5 responsive to photoexposure in a second spectral region distinct from the first spectral 
region; 

logic for selecting for each of the first pixels at least one spatially pixel 
responsive to photoexposure in a third spectral region, the third spectral region being 
distinct from the first and second spectral regions, to determine at least one associated 

10 third pixel; and 

logic for associating the intensity value of the associated second pixel and 
the intensity value of the associated third pixel with the intensity value of the first pixel to 
determine a matching set therewith, the matching set including an associated selected 
first pixel, selected second pixel and selected third pixel; and 

15 logic for determining a first gain coefficient for application to the intensity 

values of each of the pixels associated with the second spectral region based upon an 
accumulation of the intensity values associated with the selected second pixels and for 
determining a second gain coefficient for application to the intensity values of each of the 
pixels associated with the third spectral region based upon an accumulation of the 

20 intensity values associated with the selected third pixels. 
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11. A method of processing data representative of a color image based upon color 
information extracted from pixels in an imaging array, the imaging array including a 
plurality of pixels, each of the plurality of pixels being responsive to photon energy in 
one of a plurality of distinct spectral regions, each of the spectral regions being associated 

5 with one of a plurality of color channels, each of the pixels being capable of providing 
data representative of an intensity of photoexposure in the spectral region and color 
channel associated with the pixel over an exposure period, the method comprising: 

identifying white regions in the image based upon a dispersion of the 
intensities of photoexposure at a group of associated pixels in the imaging array, each of 

10 the associated pixels being responsive to photoexposure in a distinct one of the plurality 
spectral regions or color channels; and 

determining gain coefficients to be applied to intensities of photoexposure 
in the image for pixels associated with at least one of the color channels based upon an 
accumulation of the intensities of photoexposure of the pixels associated with the at least 

15 one color channel in the white regions of the image. 

12. The method of claim 1 1 , wherein the step of identifying the white regions in the 
image further comprises: 

selecting a reference channel from among the plurality of color channels; 
20 determining groups of associated pixels in the image, each of the groups 

including at least one reference channel pixel associated with the reference channel and at 
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least one non-reference channel pixel associated with a color channel distinct from the 
reference channel; and 

for each group of associated pixels, comparing an intensity of 
photoexposure of the reference channel pixel with an intensity of photoexposure of the at 
5 least one non-reference channel pixel 

1 3 . The method of claim 1 2, wherein the step of identifying the whhe regions in the 
image further comprises: 

associating first and second non-reference channel pixels with each group 
10 of associated pixels; and 

for each group of associated pixels, 

determining whether an intensity of photoexposure of the first non- 
reference chaimel pixel and an intensity of photoexposure of a second 
non-reference channel are within a predetermined range about the 
15 intensity of photoexposure of the reference channel pixel reference 

channel pixel with an intensity of photoexposure of the at least one non- 
reference channel pixel; and 

determining whether the difference between the intensities of 
photoexposure of the first and second non-reference channel pixels is less 
20 than a predetermined difference. 
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14. The method of claim 1 1 , the method further including: 

calculating intermediate gain coefficients based upon the accumulation of 
the intensities of photoexposure of the pixels associated with the at least one color 
channel in the white regions of the image; and 
5 selecting the gain coefficients to be apphed to the intensities of 

photoexposure in the image from among a plurality of sets of gain coefficients 
stored in memory based upon a closeness of the intermediate gain coefficients to 
the selected set of gain coefficients. 



15. In a camera, the camera having an imaging array, the imaging array including a 
10 plurality of pixels which are responsive to photon energy in a distinct spectral region, 
each of the pixels being capable of outputting an intensity value which is representative 
of an intensity of photoexposure in the spectral region associated with the pixel over an 
exposure period, a lens for focusing an image of an object onto the imaging array, and a 
processor, the improvement including: 
15 logic for identifying white regions in the image based upon a dispersion of 

the intensities of photoexposure at associated pixels, each of the associated pixels being 
responsive to photoexposure in a distinct one of the plurality spectral regions or color 
channels; and 

logic for determining gain coefficients to be applied to intensities of 
20 photoexposure in the image for pixels associated with at least one of the color channels 
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based upon an accumulation of the intensities of photoexposure of the pixels associated 
with the at least one color channel in the white regions of the image. 

1 6. The camera of claim 1 5, wherein the logic for identifying the white regions in the 
5 image further comprises: 

logic for selecting a reference channel from among the plurality of color 

channels; 

logic for determining groups of associated pixels in the image, each of the 
groups including at least one reference channel pixel associated with the reference 
10 channel and at least one non-reference channel pixel associated with a color channel 
distinct from the reference channel; and 

for each group of associated pixels, logic for comparing an intensity of 
photoexposure of the reference channel pixel with an intensity of photoexposure of the at 
least one non-reference channel pixel. 

15 

17. The camera of claim 15, wherein the logic for identifying the white regions in the 
image further comprises: 

logic for associating first and second non-reference channel pixels with 
each group of associated pixels; and 
20 for each group of associated pixels, 

logic for determining whether an intensity of photoexposure of the 
first non-reference channel pixel and an intensity of photoexposure of a 
second non-reference channel are within a predetermined range about the 
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intensity of photoexposure of the reference channel pixel reference 
channel pixel with an intensity of photoexposure of the at least one non- 
reference channel pixel; and 

logic for determining whether the difference between the 
5 intensities of photoexposure of the first and second non-reference channel 

pixels is less than a predetermined difference. 



18. The camera of claim 1 5, the camera further comprising: 

logic for calculating intermediate gain coefficients based upon the 
10 accumulation of the intensities of photoexposure of the pixels associated with the 

at least one color channel in the white regions of the image; and 

logic for selecting the gain coefficients to be applied to the intensities of 
photoexposure in the image from among a plurality of sets of gain coefficients 
stored in memory based upon a closeness of the intermediate gain coefficients to 
15 the selected set of gain coefficients. 



19, A computer readable medium for use in conjunction with an imaging array for 
receiving an image of an object, the imaging array including a plurality of pixels which 
are responsive to photon energy in one of a plurality of distinct spectral regions, each of 
the spectral regions being associated with one of a plurality of color channels, each of the 
20 pixels being capable of providing data representative of an intensity of photoexposure in 
the spectral region and color channel associated with the pixel over an exposure period, 
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the computer readable medium having computer readable instructions encoded thereon 
for performing the following: 

identifying white regions in the image based upon a dispersion of the 
intensities of photoexposure at associated pixels, each of the associated pixels being 
5 responsive to photoexposure in a distinct one of the plurality spectral regions or color 
channels; and 

determining gain coefficients to be applied to intensities of photoexposure 
in the image for pixels associated with at least one of the color channels based upon an 
accumulation of the intensities of photoexposure of the pixels associated with the at least 
10 one color channel in the white regions of the image. 

20. The computer readable medium of claim 19, the computer readable medium 
further including computer readable instructions encoded thereon for: 

selecting a reference channel from among the plurality of color channels; 
determining groups of associated pixels in the image, each of the groups 
including at least one reference channel pixel associated with the reference channel and at 
least one non-reference channel pixel associated with a color channel distinct from the 
reference channel; and 

for each group of associated pixels, comparing an intensity of 
photoexposure of the reference channel pixel with an intensity of photoexposure of the at 
least one non-reference channel pixel. 




I \Conimoii Data\Nagy FilesWpd\BIOMORPH\249452VPTO\uspap DOC 



45 



21 . The computer readable medium of claim 19^ the computer readable medium 
further including computer readable instructions encoded thereon for: 

associating first and second non-reference channel pixels with each group 
of associated pixels; and 
5 for each group of associated pixels, 

determining whether an intensity of photoexposure of the first non- 
reference channel pixel and an intensity of photoexposure of a second 
non-reference channel are within a predetermined range about the 
intensity of photoexposure of the reference channel pixel reference 
10 channel pixel with an intensity of photoexposure of the at least one non- 

reference channel pixel; and 

determining whether the difference between the intensities of 
photoexposure of the first and second non-reference channel pixels is less 
than a predetermined difference. 

15 

22. The computer readable medium of claim 19, the computer readable medium 
further including computer readable instructions encoded thereon for: 

calculating intermediate gain coefficients based upon the accumulation of 
the intensities of photoexposure of the pixels associated with the at least one color 
20 channel in the white regions of the image; and 

selecting the gain coefficients to be applied to the intensities of 
photoexposure in the image from among a plurality of sets of gain coefficients 
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stored in memory based upon a closeness of the intermediate gain coefficients to 
the selected set of gain coefficients. 



23. An image processor for use in conjunction with an imaging array, the imaging 
5 array including a plurality of pixels which are responsive to photon energy in one of a 
plurality of distinct spectral regions, each of the spectral regions being associated with 
one of a plurality of color channels, each of the pixels being capable of providing data 
representative of an intensity of photoexposure in the spectral region and color channel 
associated with the pixel over an exposure period, the image processor comprising: 
10 logic for identifying white regions in the image based upon a dispersion of 

the intensities of photoexposure at associated pixels, each of the associated pixels being 
responsive to photoexposure in a distinct one of the plurality spectral regions or color 
channels; and 

logic for determining gain coefficients to be applied to intensities of 
15 photoexposure in the image for pixels associated with at least one of the color channels 
based upon an accumulation of the intensities of photoexposure of the pixels associated 
with the at least one color channel in the white regions of the image. 



24. The image processor of claim 23, wherein the logic for identifying the white 
20 regions in the image further comprises: 

logic for selecting a reference channel from among the plurality of color 

chaimels; 



i \Comnion DataWagy Files\wpdVBIOMORPH\249452VPTO\uspap DOC 



47 



logic for determining groups of associated pixels in the image, each of the 
groups including at least one reference channel pixel associated with the reference 
charmel and at least one non-reference channel pixel associated with a color channel 
distinct from the reference channel; and 

for each group of associated pixels, logic for comparing an intensity of 
photoexposure of the reference channel pixel with an intensity of photoexposure of the at 
least one non-reference channel pixel. 

25. The image processor of claim 23, wherein the logic for identifying the white 

regions in the image further comprises: 

logic for associating first and second non-reference channel pixels with 

each group of associated pixels; and 

for each group of associated pixels, 

logic for determining whether an intensity of photoexposure of the 
first non-reference channel pixel and an intensity of photoexposxxre of a 
second non-reference channel are within a predetermined range about the 
intensity of photoexposure of the reference channel pixel reference 
channel pixel with an intensity of photoexposure of the at least one non- 
reference charmel pixel; and 

logic for determining whether the difference between the 
intensities of photoexposure of the first and second non-reference channel 
pixels is less than a predetermined difference. 
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26. The image processor of claim 23, the image processor further including: 
logic for calculating intermediate gain coefficients based upon the 
accumulation of the intensities of photoexposure of the pixels associated with the 
at least one color channel in the white regions of the image; and 
5 logic for selecting the gain coefficients to be applied to the intensities of 

photoexposure in the image from among a plurality of sets of gain coefficients 
stored in memory based upon a closeness of the intermediate gain coefficients to 
the selected set of gain coefficients. 



10 27. A method of processing data representative of a color image based upon color 
information extracted from pixels in an imaging array, the imaging array including a 
plurality of pixels, each of the plurality of pixels being responsive to photon energy in 
one of a plurality of distinct spectral regions, each of the spectral regions being associated 
with one of a plurality of color channels, each of the pixels being capable of providing 
15 data representative of an intensity of photoexposure in the spectral region and color 
channel associated with the pixel over an exposure period, the method comprising: 

associating a group of pixels in the array, the group of pixels including at least 
one pixel associated with each of the plurality of color channels; 

for each of the pixels in the associated group of pixels, determining an associated 
20 whiteness weight based upon a dispersion among the intensities of photoexposure at 
pixels in the group associated with distinct ones of the color channels; and 

determining gain coefficients to be applied to intensities of photoexposure in the 
image for pixels associated with at least one of the color channels based upon an 
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accumulation of the intensities of photoexposure of each of the pixels of the color 
channel weighted by the whiteness weight associated with the pixel. 

28. The method of claim 27, the method further including determining an associated 
5 whiteness weight using fuzzy logic. 

29. The method of claim 27, the method further including determining a gain 
coefficient to be applied to the pixel intensity values of one of the color channels based 
upon an accumulation of at least some of the pixel intensity values of the color charmel 

10 weighted by corresponding whiteness weights associated with the pixel intensity values 
of the color channel. 

30. In a camera, the camera having an imaging array, the imaging array including a 
plurality of pixels, each of the plurality of pixels being responsive to photon energy in 

15 one of a plurality of distinct spectral regions, each of the spectral regions being associated 
with one of a plurality of color channels, each of the pixels being capable of providing 
data representative of an intensity of photoexposure in the spectral region and color 
channel associated with the pixel over an exposure period, the improvement including: 

logic for associating a group of pixels in the array, the group of pixels including at 
20 least one pixel associated with each of the plurality of color channels; 

for each of the pixels in the associated group of pixels, logic for determining an 
associated whiteness weight based upon a dispersion among the intensities of 
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photoexposure at pixels in the group associated with distinct ones of the color channels; 
and 

logic for determining gain coefficients to be applied to intensities of 
photoexposure in the image for pixels associated with at least one of the color channels 
5 based upon an accumulation of the intensities of photoexposure of each of the pixels of 
the color channel weighted by the whiteness weight associated with the pixel. 

3 1 . The camera of claim 30, the improvement further including logic for determining 
an associated whiteness weight using fuzzy logic. 

10 

32. The camera of claim 30, the improvement further including logic for determining 
a gain coefficient to be applied to the pixel intensity values of one of the color channels 
based upon an accumulation of at least some of the pixel intensity values of the color 
channel weighted by corresponding whiteness weights associated with the pixel intensity 

1 5 values of the color channel . 

33. An image processor for use in conjunction with an imaging array, the imaging 
array including a plurality of pixels, each of the plurality of pixels being responsive to 
photon energy in one of a plurality of distinct spectral regions, each of the spectral 

20 regions being associated with one of a plurality of color channels, each of the pixels 
being capable of providing data representative of an intensity of photoexposure in the 
spectral region and color channel associated with the pixel over an exposure period, the 
image processor including: 
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logic for associating a group of pixels in the array, the group of pixels including at 
least one pixel associated with each of the plurality of color channels; 

for each of the pixels in the associated group of pixels, logic for determining an 
associated whiteness weight based upon a dispersion among the intensities of 
5 photoexposure at pixels in the group associated with distinct ones of the color channels; 
and 

logic for determining gain coefficients to be applied to intensities of 
photoexposure in the image for pixels associated with at least one of the color channels 
based upon an accumulation of the intensities of photoexposure of each of the pixels of 
10 the color channel weighted by the whiteness weight associated with the pixel. 

34. The image processor of claim 33, the image processor further including logic for 
determining an associated whiteness weight using fuzzy logic. 

15 35. The image processor of claim 33, the image processor further including logic for 
determining a gain coefficient to be applied to the pixel intensity values of one of the 
color channels based upon an accumulation of at least some of the pixel intensity values 
of the color channel weighted by corresponding whiteness weights associated with the 
pixel intensity values of the color channel. 

20 

36. A computer readable medium for use in conjuction with processing data 
representative of a color image based upon color information extracted from pixels in an 
imaging array, the imaging array including a plurality of pixels, each of the plurality of 
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pixels being responsive to photon energy in one of a plurality of distinct spectral regions, 
each of the spectral regions being associated with one of a plurality of color channels, 
each of the pixels being capable of providing data representative of an intensity of 
photoexposure in the spectral region and color channel associated with the pixel over an 
5 exposure period, the computer readable medium having computer readable instructions 
encoded thereon for: 

associating a group of pixels in the array, the group of pixels including at least 
one pixel associated with each of the plurality of color channels; 

for each of the pixels in the associated group of pixels, determining an associated 
10 whiteness weight based upon a dispersion among the intensities of photoexposure at 
pixels in the group associated with distinct ones of the color channels; and 

determining gain coefficients to be applied to intensities of photoexposure in the 
image for pixels associated with at least one of the color channels based upon an 
accumulation of the intensities of photoexposure of each of the pixels of the color 
15 channel weighted by the whiteness weight associated with the pixel 

37. The computer readable medium of claim 36, the computer readable medium 
further including computer readable instructions encoded thereon for determining an 
associated whiteness weight using fuzzy logic. 

20 

38. The computer readable medium of claim 36, the computer readable medium 
further including computer readable instructions encoded thereon for determining a gain 
coefficient to be applied to the pixel intensity values of one of the color channels based 
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upon an accumulation of at least some of the pixel intensity values of the color channel 
weighted by corresponding whiteness weights associated with the pixel intensity values 
of the color channel. 
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ABSTRACT 

A system and method for processing data representative of a color image is 
disclosed. Image data represents an intensity of photoexposure of an imaging array at 
specific locations in the imaging array and in distinct spectral regions corresponding to 
color channels. A process identifies "white" regions in the color image by comparing the 
intensities of photoexposure of groups of associated pixels which are responsive to 
photon energy in different spectral regions. If the intensities of photoexposure of the 
pixels in the group of associated pixels are proportionally equivalent, these pixels are 
determined to be in a white region of the image. White balancing gain coefficients are 
then based upon the pixel intensity values at pixel locations in the white regions of the 
image. 
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APPLICATION NO. Date 



I hereby claim the benefit under Title 35, United States Code, Section 120 of any United States 
applicatlon(s) listed below and, insofar as the subject matter of each of the claims of this application is 
not disclosed in the prior United States application In the manner provided by the first paragraph of 
Title 35, United States Code, Section 112, I acknowledge the duty to disclose all information known to 
me to be material to patentability as defined in Title 37, Code of Federal Regulations, Section 156 
which became available between the filing date of the prior application and the national or PCT 

international filing date of this application: 

Status -(patented, 

APPLICATION NO. Filing Date pending, abandoned) 
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Paul N. Kokulis, Reg. No. 16773; Raymond F. Lippitt, Reg. No. 17519; G. Lloyd Knight, Reg. No. 17698; 
Carl G Love Reg. No. 18781; Kevin E. Joyce, Reg. No. 20508; George M. Sirilla, Reg. No. 18221; Donald J. 
Bird Reg. No. 25323; Peter W. Gowdey, Reg. No. 25872; Dale S. Lazar, Reg. No. 28872; Paul E. White, Jr., 
Reg No 32011; Glenn J. Perry, Reg. No. 28458; Kendrew H. Colton, Reg. No. 30368; G. Paul Edgell, Reg. No. 
24238- Lynn E. Eccleston, Reg. No. 35861; Timothy J. Klima, Reg. No. 34852; David A. Jakopin, Reg. No. 
32995- Mark G. Paulson, Reg. No. 30793; Stephen C. Glazier, Reg. No. 31361; Paul F. McQuade, Reg. No. 
31542- Ruth N. Morduch, Reg. No. 31044; Richard H. Zaitlen, Reg. No. 27248; Roger R. Wise, Reg. No. 31204; 
Jay m' Finkelstein, Reg. No. 21082; Anita M. Kirkpatrick, Reg. No. 32617; Michael R. Dzwonczyk, Reg. No. 
36787; W. Patrick Bengtsson, Reg. No. 32456; Jack S. Barufka, Reg. No. 37087; Paul G. Nagy, Reg. No. 
37 896; Steven W. Smyrski, Reg. No. 38,312; Eric S. Chen, Reg. No. 43,542; Vivian S. Shin, Reg. No. 43,919; 
Adam R. Hess, Reg. No. 41835; my patent attorneys of PILLSBURY MADISON & SUTRO LLP, with offices located 
at 725 South Figueroa Street, Suite 1200, Los Angeles, California, 90017 telephone (213) 488-7100, and 



Send correspondence to Mr. Paul G. Nagy, PILLSBURY MADISON & SUTRO LLP, 725 South Figueroa 
Street, Suite 1200, Los Angeles, California, 90017, and direct telephone calls to Mr. Paul G. Nagy, 
(213) 488-7105. 



1 


Fulf Name of Sole/First InvenTdr: 
THOMAS EUGENE HUBINA 


Residence: 

SIMI VALLEY CALIFORNIA 


Citizenship: 
USA 


(Citv^ (State) 


Post Office Address: 

1718 Sinaloa Road, #117, Simi Valley, California 93065 


2 


Full Name of Sole/Second Inventor: 
HRAin CHARLES REINHART 


Residence: 

MOORPARK CALIFORNIA 


Citizenship: 
USA 


(CWw) (State) 


Post Office Address: 

12967 Silver Creek Street, Moorpark, California 93021 


3 


Full Name of Sole/Third Inventor: 


Residence: 


Citizenship: 


(Citv) (State) 


Post Office Address: 


4 


Full Name of Sole/Fourth Inventor: 


Residence: 


Citizenship: 


(Citv) (State) 


Post Office Address: 



I hereby declare that all statements made herein of my own knowledge are true and that all statements 
made on information and belief are believed to be true; and further that these statements were made 
with the knowledge that willful false statements and the like so made are punishable by fine or 
Imprisonment, or both, under Section 1001 of Title 18 of the United States Code and that such willful 
false statements may jeopardize the validity of the application or any patent issued thereon. 



ThomarE^^^H lici^a^ 


Craiq Chfarles Reinhart 


Date ^-jy-Jlovo 


Date f//B/zor>0 
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